System and method for dynamic three dimensional command and control

ABSTRACT

An exemplary system for monitoring an operational area, includes a processor that receive plural data streams. Each data stream can include a different spatial characteristic of the operational area. The processor also generates a three-dimensional (3D) virtual visualization of the operational area based on observational perspectives associated with the data streams and their associated spatial characteristics and dynamically prioritizes operational sub-regions within the operational area based on the spatial characteristics. The processor generates a signal encoded with data for verifying the 3D virtual visualization of the operational area including the prioritized operational sub-regions.

FIELD

The present disclosure relates to systems and methods for monitoring anoperational area.

BACKGROUND

It is often desirable to maintain situational awareness of a forceprotection area to mitigate hostile actions against military personnel,the general public, resources, facilities and critical information, suchas a military base, airport, stadium, transit system, smart city, powerplant or detention facility, and autonomously react to newly sensedinformation in these locations. In today's command and controlenvironment, rapid reaction to newly acquired information is necessaryso an operator can decide on actions to quickly changing circumstances.

An operator can quickly make decisions by having the system recommendpredefined rules of engagement based on the newly sensed situationalawareness information. Further, an autonomous system may be able tosupport decision-making actions independent of an operator if theautomated responses are properly controlled and potential consequencesof improper responses are carefully considered and the associatedpotential risks are deemed to be warranted.

Sensor fusion systems demonstrate the efficacy of combining diverseperspectives in an operational area to improve decision makingprocesses. Such systems may provide insight to facilitate objectdetection, identification, classification, geospatial positioning,geospatial navigation, and many other contextual data metrics that canbe derived from data of heterogeneous sensors. With a potential datavolume greater than what can be analyzed by unaided human intellect,computer-based support can be necessary, or at a minimum facilitate, theprocessing and analysis of new data. Further, desired insights mayrequire additional data or analytic processing—such as making sensoradjustments, performing rapid analytics of the data, or real-timereaction to immediate threats that are identified as a result of dataanalysis.

SUMMARY

An exemplary system for monitoring an operational area is disclosed, thesystem comprising: a processor configured to: receive plural datastreams, each data stream including a different spatial characteristicof the operational area; generate a three-dimensional (3D) virtualvisualization of the operational area based on observationalperspectives associated with the data streams and their associatedspatial characteristics; dynamically prioritize operational sub-regionswithin the operational area based on the spatial characteristics; andgenerate a signal encoded with data for verifying the 3D virtualvisualization of the operational area including the prioritizedoperational sub-regions.

An exemplary method for monitoring an operational area is disclosed, themethod comprising: in one or more devices connected in a network:receiving one or more data streams, each data stream including adifferent spatial characteristic of the operational area; generating a3D virtual visualization of the operational area based on anobservational perspective associated with each data stream and thedifferent spatial characteristics included in each data stream;dynamically prioritizing operational sub-regions within the operationalarea based on the received spatial characteristics; and generating asignal for verifying the 3D visual virtualization of the operationalarea including the prioritized operational sub-regions.

An exemplary computer readable medium encoded with a program forperforming a method of monitoring an operational area, which when placedin communicative contact with a processor is disclosed, the programconfiguring the processor to: generate a three-dimensional (3D) virtualvisualization of the operational area based on an observationalperspective of plural data streams and different spatial characteristicsincluded in each data stream; dynamically prioritize operationalsub-regions within the operational area based on the spatialcharacteristics; and generate a signal for verifying the 3D virtualvisualization of the operational area including the prioritizedoperational sub-regions.

BRIEF DESCRIPTION OF THE DRAWINGS

Exemplary embodiments are best understood from the following detaileddescription when read in conjunction with the accompanying drawings.Included in the drawings are the following figures:

FIG. 1 illustrates a system for dynamic three-dimensional command andcontrol in accordance with an exemplary embodiment of the presentdisclosure.

FIG. 2 illustrates an exemplary computing device and edge detectiondevices in accordance with an exemplary embodiment of the presentdisclosure.

FIG. 3 illustrates a 3D virtual visualization of an operational area inaccordance with an exemplary embodiment of the present disclosure.

FIGS. 4A-4C illustrate a flow diagram of a method performed by thesystem of FIG. 1 in accordance with an exemplary embodiment of thepresent disclosure.

Further areas of applicability of the present disclosure will becomeapparent from the detailed description provided hereinafter. It shouldbe understood that the detailed description of exemplary embodiments isintended for illustration purposes only and is, therefore, not intendedto necessarily limit the scope of the disclosure.

DETAILED DESCRIPTION

Exemplary embodiments of the present disclosure can provide real-timethree-dimensional (3D) situational awareness of morphological objects inan operational area and support command, control, communications,collection of intelligence, analysis of data regarding the operationalarea, surveillance, and reconnaissance (C4ISR). In addition, thedisclosed exemplary embodiments can instantiate efficient means forprocessing heterogeneous sensor data and control dynamic sensor datacollection without human intervention. The system and methods describedherein extend beyond common sensor fusion methods by interrogatingobjects after initial sensor data ingestion occurs. For example, thedisclosed system and methods can be implemented in the control of servomotors (e.g., optical pan/tilt/zoom), steering electromagnetic antennas(e.g., phased array antennas, servo motor-controlled directionalantennas, operating plasma antennas, or importing interferometry oflaser analysis of cesium atoms), software defined radios (e.g.,controlled adjustment of spectral monitoring), and relocation ofunmanned vehicles equipped with sensors to other positions within andoutside of the operational area.

Additionally, the exemplary embodiments disclosed herein may providecapabilities for autonomously targeting, exploiting and defeatingreal-time threats to an operational area if those threats and respectiverules of engagement have been confirmed. Exploiting or defeating targetsmay include soft kill weapons such as electronic warfare tools or hardkill weapons such as directed energy equipment and traditional kineticsystems.

The exemplary embodiments of the present disclosure provide severaladvantages including: Emergency Response for situational awareness inlocating and tracking resources, threats, and military personnel orcivilians in need of assistance; automated robotics surroundingawareness including use cases such as warehouse operations andself-driving vehicles; and in assistance with off the grid navigationservices in the event satellite, triangulation, or other means ofidentifying spatial coordinates are unavailable

FIG. 1 illustrates a system for dynamic three-dimensional command andcontrol in accordance with an exemplary embodiment of the presentdisclosure. FIG. 2 illustrates an exemplary computing device and edgedetection devices in accordance with an exemplary embodiment of thepresent disclosure.

As shown in FIGS. 1 and 2, an exemplary system 100 for monitoring anoperational area 110 is disclosed. The system 100 includes a computingsystem 120 having a communication interface 122, a processor 124, aninput/output (I/O) interface 126, a memory device 128, and a displaydevice 130. The components of the computing system 120 can include acombination of hardware and software devices. The computing system 120can be configured to: receive plural data streams via the communicationinterface 122, each data stream including a different spatialcharacteristic of the operational area 110. The processor 124 can beconfigured to generate a three-dimensional (3D) virtual visualization ofthe operational area for display on the display device 130 based onobservation perspectives associated with the data streams and thespatial characteristics. The processor 124 can also be configured todynamically prioritize operational sub-regions within the operationalarea 110 based on the spatial characteristics and generate a signalencoded with data for verifying the three-dimensional virtualvisualization of the operational area 110 including the prioritizedoperational sub-regions.

The communication interface 122 can include a receiving and transmittingdevice configured to connect to a network 140. The communicationinterface 122 can be encoded with program code to receive and transmitdata signals and/or data packets over the network 140 according to aspecified communication protocol and data format. During a receiveoperation, the communication interface 122 can identify parts of thereceived data via the header and parse the data signal and/or datapacket into small frames (e.g., bytes, words) or segments for furtherprocessing at the processor 124. During a transmit operation, thecommunication interface 122 can receive data from the processor 124 andassemble the data into a data signal and/or data packets according tothe specified communication protocol and data format of the network 140.The communication interface 122 can include one or more receivingdevices and transmitting devices for providing data communicationaccording to any of a number of communication protocols and data formatsas desired. For example, the communication interface 122 can beconfigured to communicate over the network 140, which may include alocal area network (LAN), a wide area network (WAN), a wireless network(e.g., Wi-Fi), a mobile communication network, a satellite network, theInternet, optic fiber, coaxial cable, infrared, radio frequency (RF), orany combination thereof. Other suitable network types and configurationswill be apparent to persons having skill in the relevant art. Thecommunication interface 122 can include any suitable hardware componentssuch as an antenna, a network interface (e.g., an Ethernet card), acommunications port, a PCMCIA slot and card, or any suitable processingdevices for performing functions according to the exemplary embodimentsdescribed herein.

The processor 124 can be a special purpose or a general purpose hardwareprocessing device encoded with program code or software for performingthe exemplary embodiments disclosed herein. The processor 124 can beconnected to a communications infrastructure including a bus, messagequeue, network, multi-core message-passing scheme, etc. The processor124 can include one or more processing devices such as a microprocessor,central processing unit, microcomputer, programmable logic unit or anyother suitable hardware processing device as desired.

The I/O interface 126 can be configured to receive the signal from theprocessor and generate an output verifying the 3D virtual visualizationof the operational area. The I/O interface 126 can include a combinationof hardware and software for example, a processor, a circuit card, orany other suitable hardware device encoded with program code, software,and/or firmware for communicating with a peripheral device such as thedisplay device 130.

The memory device 128 can be configured to store the plural datastreams. The memory device 128 can include one or more memory devicessuch as volatile or non-volatile memory. For example, the volatilememory can include random access memory, read-only memory, etc. Thenon-volatile memory can include a resident memory device such a harddisk drive and a removable storage drive (e.g., a floppy disk drive, amagnetic tape drive, an optical disk drive, a flash memory, or any othersuitable device). The non-volatile memory can also or in the alternativeinclude an external memory device connected to the computing device 102via the I/O interface 126. Data stored in the computer system 120 (e.g.,in a non-volatile memory) may be stored on any type of suitable computerreadable media, such as optical storage (e.g., a compact disc, digitalversatile disc, Blu-ray disc, etc.) or magnetic tape storage (e.g., ahard disk drive). The data may be configured in any type of suitabledatabase configuration, such as a relational database, a structuredquery language (SQL) database, a distributed database, an objectdatabase, etc. Suitable configurations and storage types will beapparent to persons having skill in the relevant art.

The display device 130 can include high-definition multimedia interface(HDMI), digital visual interface (DVI), video graphics array (VGA), orany suitable display device or display type as desired. The display 130may be any suitable type of display for displaying data transmitted viathe I/O interface 126 of the computer system 120, including a cathoderay tube (CRT) display, liquid crystal display (LCD), light-emittingdiode (LED) display, capacitive touch display, thin-film transistor(TFT) display, etc.

The operational area 110 can include a sensor arrangement 112 configuredto generate the plural data streams by observing the operational area.The sensor arrangement 112 can include a plurality of heterogeneoussensors DS1-DSn of various types such as sensors any one or more of anacoustic, radio frequency, electro-optical, thermal, chemical,biological, radiological, nuclear, explosive, temperature, mechanical,etc. Each sensor DS1-DSn can be attached, affixed, or integrated into anedge device 114 to establish a dynamic sensor (DS) or edge device thatcan be moved between two or more locations within the operational area110. The sensor data is dynamically acquired, meaning one or more of thedynamic sensors D1-DSn may appear or disappear on the network 140 at anygiven time. The sensor data is then stored in a memory device 117 orreal-time memory-based database 119, such as Redis or any suitabledatabase as desired, for later processing and analysis operations. Theedge device 114 with an attached or integrated dynamic sensor DSn caninclude a communication interface 116 configured to receive the pluraldata streams of other sensors. The edge device 114 can include one ormore processors 118 configured to generate a 3D virtual visualization ofthe operational area 110, dynamically prioritize operational sub-regionswithin the operational area 110, and generate a signal encoded with datafor verifying the 3D virtual visualization of the operational area 110.Each dynamic sensor DSn is configured to transmit a data streamperiodically or non-periodically to the computing device 120 over thenetwork 140. For example, one or more dynamic sensors DS1-DSn can beconfigured to transmit data to the computing device 120 when a change ina spatial characteristic of an operational sub-region is detected. Eachdata stream has data of a sensor type and is generated by thecorresponding sensor observing the operational area, and a 3D geometryof each operational sub-region is created using one or morecharacteristics of the sensor arrangement. For example, if the dynamicsensor DSn of an edge device 114 is an image sensor or camera, the 3Dgeometry of the operational sub-region 310 can be determined by a fieldof view (e.g., a cone or triangle) of the image. According to anotherexemplary embodiment, the dynamic sensor DSn can be an acoustic sensorwherein the sensing field can be represented by a circular shape.According to another exemplary embodiment, the 3D geometry of eachoperational sub-region 310 can be dimensioned according to spatialproperties of the operational area 110. For example, one or moreoperational sub-regions 115 can include a 3D geometry having sidesand/or a shape limited or constrained by the bordering features (e.g.,bodies of water) or physical features (e.g., mountainous or rockyterrain) of the operational area.

FIG. 3 illustrates a 3D virtual visualization of an operational area inaccordance with an exemplary embodiment of the present disclosure. The3D virtual visualization of the operational area 110 generated by thecomputer system 120 and the one or more processors 118 of the edgedevices 114 includes “3D grids” that define plural sub-regions 320within the operational area 110 extended to a preferred ceiling height305 above the terrestrial floor 315. For example, each of the computersystem 120 and the one or more processors 118 of the edge devices 114 isconfigured to generate the 3D virtual visualization of the operationalarea by dividing a space of the operational area 110 into a plurality of3D grid spaces 310. Each operational sub-region 320 includes one or more3D grid spaces 310. Each 3D space 310 is bounded by ground-levelaltitude 325 (e.g., the height above ground level), a user-definedceiling altitude 305 (e.g., the highest altitude at which the edgedevice 114 or dynamic sensors DS1-DSn can reach in the operational area110), and the jurisdictional terrestrial boundary requiringprotection/monitoring 330 (e.g., land-based geographical areas). Thisphysical space may dynamically change if one or more of the dynamicsensors DS1-DSn detect a new activity or a change in a spatialcharacteristic of an operational sub-region 320 (e.g., a vehicle'spresence and its on-board sensors may define a new geographic area tohave relevant importance).

The computer system 120 and the one or more processors 118 of the edgedevices 114 can be configured to: assign a real-time weighting value toeach operational sub-region based on one or more of: an importance to aspecified operation, a spatial or temporal relevance to a specifiedoperation, time since last sensor observation, available granularity ofthe spatial characteristics in the data streams, observationalperspectives of sensors from which the data streams are generated withrespect to an operational sub-region, corresponding sensor-types, orvelocities of anomalies within a current or adjacent operationalsub-region. Based on the time since a last sensor observation of one ormore sensors that are observing an operational sub-region, the computersystem 120 and the one or more processors 118 of the edge devices 114can be configured to increase a priority of the operational sub-regionwhen an interval between receptions of data streams from a sensorarrangement of the operational sub-region increases. For example, as theintervals between sensor scans of the operational sub-region 310increase, its weight increases so a dynamic sensor DSn system will bedirected to observe the operational sub-region 310 in the next availableopportunity.

The computer system 120 and the one or more processors 118 of the edgedevices 114 can be configured to assign a priority to an operationalsub-region 320 based on proximity of a sensor arrangement to a detectedanomaly. For example, sensor proximity can be weighted to where one ormore distant dynamic sensors among DS1-DSn may continuously observe anobject, but a dynamic sensor of DS1-DSn in closer proximity may warrantadditional observation of an operational sub region 320 to ensureartifacts are not overlooked. The weight of an operational sub region320 increases over time if any dynamic sensors among DS1-DSn fromdifferent observational perspectives do not observe and transmit datastreams regarding the operational sub-regions 320. This process allowsthe observational perspectives of an operational area or sub-region 320to be diversified with respect to nature/lighting/weather/etc. and anyman-made or natural obstructions. Observational perspective is notentirely objective, as details and analysis results can varysubstantially based on perspective. According to an exemplaryembodiment, compass rose-based quadrants can be assigned to one or moreoperational sub regions 320 and dynamic sensors DS1-DSn to define and/orestablish differentiation in perspective. For example, dynamic sensorsDS1-DSn from a first perspective may encounter sunlight, which caninterfere with object views in a specified operational sub-region 320.Those of the dynamic sensors DS1-DSn having different observationalperspectives of the same operational sub-region 320, for example fromdifferent viewpoints or viewpoints that are not impacted by thesunlight, can be instructed by the computing system 120 or another ofthe dynamic sensors DS1-DSn to inspect the specified operationalsub-region(s) 320. According to an exemplary embodiment of the presentdisclosure, sensor types will be assigned to each operational sub regionto optimally utilize dynamic sensors DS1-DSn to obtain diverse datastreams from sensors that cover the three-dimensional operational area.For example, a dynamic sensor DSn configured as a software defined radio(SDR) may by instructed and/or controlled to continuously monitor theoperational area 110 for signals at 2.4 GHz and triangulate any sensedsignals. Alternatively, any dynamic sensors DS1-DSn that are configuredas visual optical sensors in the operational area 110 as well as anydynamic sensors DS1-DSn that are configured as thermal cameras may needto view the operational sub-region (e.g., 3D grid) from many differentangles to maintain continuous monitoring. Each sensor deployed to coverthe operational area 110 is configured by the computing system 120 tohave its own weighting criteria paired with respective geographic, time,range, and perspective criteria based on sensor type and data streamcontent.

According to an exemplary embodiment, the computer system 120 and theone or more processors 118 of the edge devices 114 can be configured toactively weight each sub region by comparing the 3D geometries of therespective sub regions to the geometries of the representativegeometries of each deployed sensor and associated sensor type. Forexample, a sphere of a certain size may denote the receive sensitivityof a dynamic sensor DSn having an omni-directional radio frequencyantenna. As another example, a pyramid may denote the real-timepositioning of a security camera field of view.

According to an exemplary embodiment, the computer system 120 and theone or more processors 118 of the edge devices 114 can be configured todecrease a priority of an operational sub-region over time when datastreams received from the operational sub-region 320 are acquired by twoor more sensors from a common observational perspective. As the geometryof a dynamic sensor DSn overlaps a sub-region geometry, the computingsystem 120 and or the one or more processors of other dynamic sensorsamong DS1-DSn can confirm overlap to denote coverage is met. As a resultof the overlap in coverage, the computing system 120 can relax thereal-time weighting criteria of that sub-region. As the real-timeweighting dynamically changes, the computing device 110 or processor ofanother dynamic sensor DSn can direct sensors with dynamic properties(e.g., pan-tilt-zoom motors) to change position and obtain additionalinputs of the next highest priority sub region.

According to an exemplary embodiment, the computer system 120 and theone or more processors 118 of the edge devices 114 can be configured to:assign on a case-by-case basis: a first priority to an operationalsub-region 320 determined to have a relevant observational perspectiveof the operational area of a security operation; a null priority to anoperational sub-region 320 determined to lack a relevant observationalperspective of the observational area; and a second priority lower thanthe first priority to an operational sub-region 310 determined to be ona fringe of the operational area and/or undetectable by a sensor. Forexample, if the operational sub-region 320 is critical to securityoperations, a high priority is assigned. If an operational sub-region320 is outside observational areas (e.g., inside a building, belowground, or otherwise requested not to be viewed), a null priority isassigned. If the operational sub-region 320 is on or at the fringe ofthe operational area and unreachable by any of the dynamic sensorsDS1-DSn, the operational sub-region is assigned a low to null priorityrelative to other operational sub-regions in the operational area.

According to another exemplary embodiment of the present disclosure, thecomputing system 120 can be configured to assign a priority to one ormore operational sub-regions 320 or one or more dynamic sensors amongDS1-DSn having an observational perspective of an operational sub-region320 based on requirements of a specified operation, wherein at least twoor more of the assigned priorities are different. For example, one ofthe operational sub-regions 320 where vehicles and/or troops are beingassembled for an exercise or deployment may be given a higher prioritythan an operational sub-region 320 where buildings are beingconstructed.

According to an exemplary embodiment, the computer system 120 and theone or more processors 118 of the edge devices 114 can be configured toprocess each data stream by detecting one or more of: the presence of ananomaly in the operational area, a position of the anomaly in anoperational sub-region, movement of the anomaly in an operationalsub-region 320. The computing device 120 is configured to: extractportions of the spatial characteristics from each data stream andcombine the extracted portions into a combined data set and identify oneor more patterns in the combined data set.

According to an exemplary embodiment, the sensor arrangement can includetwo or more sensors having observational perspectives in eachoperational sub-region 320 and the system is configured to: prioritizeeach sensor based on characteristics including a location of theoperational sub-region, time between data acquisitions, range from ananomaly in the operational sub-region 320, or an observationalperspective within the operational sub-region 320.

According to yet another exemplary embodiment the sensor arrangement caninclude two or more sensors having observational perspectives in eachoperational sub-region 320 and the system is configured to: prioritizeeach sensor based on characteristics including a location of theoperational sub-region 310, time between data acquisitions, range froman anomaly in the operational sub-region 320, or an observationalperspective on the anomaly that is at that time located within theoperational sub-region 320.

According to another exemplary embodiment, the sensor arrangement caninclude a first sensor arrangement and a second sensor arrangement, andthe computing system 120 can be configured to: dynamically adjust thedetermined priority of the first operational sub-region, adjust one ormore properties of the first sensor arrangement based on the adjustedpriority, and acquire data from the second sensor arrangement in a nexthighest priority operational sub-region 320. The computing system 120can be configured to adjust one or more properties of the first sensorarrangement in the first operational sub-region to eliminateobservational gaps in coverage of the first operational sub-region 320,establish granular observation of the first operational sub-region 320,or perform a triangulation of an anomaly under observation in the firstoperational sub-region 320. The computing system 120 can be configuredto: identify a first sensor of the first sensor arrangement havingobservational perspective of the first operational sub-region 320 thatis currently engaged in a first observation activity, and identify asecond sensor in the first sensor arrangement that is available toengage in a second observation activity of the first operationalsub-region 320. For example, in parallel operation with sensorscollecting sensor data and communicating associated data streams, thecomputing system 120 can be configured to perform resource scheduling ofall available, potentially relevant dynamic sensors DS1-DSn (under itscontrol) to reposition them in an effort to diminish temporal andgeographic observational gaps, improve granularity of observation, orperform triangulation of a target under observation. This action caninvolve issuance of Internet Protocol (IP)-based communication topan-tilt servo motors to “cue and slew” sensors to improve the real-timecollection of data. This type of action may also require moreextravagant control of robotic (autonomous) vehicles to inspect subregions. As part of this resource scheduling, the computing system 120will be aware of sensors that were previously engaged in observationactivities where a dynamic sensor DSn has detected anomalistic behaviorand is deemed busy or occupied in an activity. The computing system 120will then search for other available sensors to support operationalneeds, where the system uses a methodology for managing sensor resourcesso that high target accuracy can be attained. The method considersavailable resources, the prioritization of operational sub-regions, andthe prioritization of targets.

According to an exemplary embodiment of the present disclosure, thecomputer system 120 can be configured to: respond to threats bydeploying resources and engaging in activities to perform predefinedcritical mitigation protocols such as, for example, responding to denialof service attacks, attacks that impair or stop functionality oroperation of the computer system or associated devices or networks. Thesystem 100 can be combined with the one or more subsystems, each ofwhich includes an electronic warfare system, a directed energy system,or a kinetic weapon system. For example, beyond relaxing operational subregion 320 prioritization, computer system or edge device software mayalso control and automate functions of other unique systems such asElectronic Warfare technologies, Directed Energy, and kinetic weapons.

FIGS. 4A-4C illustrate a flow diagram of a method performed by thesystem of FIG. 1 in accordance with an exemplary embodiment of thepresent disclosure.

As shown in FIG. 4B, the computing system 120 receives real-timedatastreams from one or more dynamic sensors (DS1-DSn) having anobservational perspective of the operational area 110 (step 402). Thecomputing system 120 is configured to merge the heterogeneous data ofthe different sensors (and sensor types) via an analysis of the shapesof the sensor fields to triangulate, track, and target an object in theoperational area 110 (step 404). As a result of the merge, the computingsystem 120 determines whether there are any overlapping shapes of anydynamic sensors among DS1 to DSn to triangulate (step 406). If there areno shapes to triangulate, the processing ends (step 408). On the otherhand, if there are shapes to triangulate, then the computing system 120triangulates the shapes (step 410). Once the shapes are processed, thecomputing system 120 can send the processed data to the communicationinterface 122 so that the data can be formatted into a control and/ordata signal and communicated to one or more external systems 150 forsituational awareness or alerting over a network 140 (step 412). Thedata signal can be used to perform various operations including, amongothers, sensor data fusion analytics where the data signal can beencoded with real-time shapefiles which can be processed to obtain dataindicating a distance of an object from sensitive locations,course/speed/heading of an object, movement behavior of an object suchas natural or mechanical drive, and assessment of potential threats toan object or an operational area 330 (step 414), which can be used by anexternal or third party system 150 (step 416) to react to or address thethreat.

Following or in parallel with the sensor data fusion operation (step404), the computing system 120 determines whether there are anyanomalies to interrogate (step 418). If there are, then the computingsystem 120 initiates command/control operations with one or more of thedynamic sensors DS1-DSn of the sensor arrangement (step 420). Thecommand/control operation can involve the computing system 120 sending“cue and slew” controls to any of the dynamic sensors with, for example,pan-tilt-zoom features, so that the operational area 330 or operationalsub-regions 320 can be interrogated to detect distant objects. Ifdistant objects are not detected, then the computing system 120 candefine a 3D operational area by evenly spacing virtual boxes across aterrain flow to an operational ceiling 305 (step 422). The computingsystem 120 establishes a real-time queue in which all operationalsub-regions within the operational area grid are listed to identifyunderserved operational sub-regions (step 424). If any underservedsub-regions are identified (step 426), then the computing system 120initiates command/control operations as discussed above (step 420). Inorder to undertake the command/control operations, the computing system120 generates data/control signals to request additional sensor datafrom one or more sensors DS1-DSn in the sensor arrangement (step 428).If there are no underserved sub-regions, the computing system 120 movesto the next box in the queue to determine if the correspondingsub-region is an underserved sub-region (step 424). Steps 424 and 426are repeated until all boxes in the list are processed.

As shown in FIG. 4A, computing system 120 can send to a dormant edgesensor DSn of the sensor arrangement a request for additional sensordata, including “cue and slew” commands to orient the sensor in adesired direction for monitoring an operational sub-region to detect anobject or activity (step 430). The processor of the dynamic sensor DSndetermines whether a pixel morphology (e.g., comparison of eachindividual pixel values with neighboring pixel values) is greater than athreshold (step 432). If the morphology exceeds the threshold, then theprocessor performs object detection within areas of detected movement byexecuting one or more algorithms from a defined list of hierarchicalmodels (step 434). In addition, a 3D bounding box is created around thearea of pixel morphology (e.g., around the object) (step 436). If anobject matching any model class above a % threshold is detected (step438), then another (e.g., second) 3D bounding box is created for theobject (436), object recognition is performed to identify uniqueidentifiers of the object (step 440), and pattern tracking is performedon the object for a duration of sensory observation (step 442).Following object recognition at step 440, it is determined whether thedetected object matches any class above a percent threshold (444). Ifthe object does match such a class, then the pattern tracking of step442 is performed and another (e.g., third) 3D bounding box is generated(step 436). Following the pattern tracking (step 442), another (e.g.,fourth) bounding box is created (step 436). Once the bounding box(es)is/are created, the dynamic sensor DSn communicates 3D vertices to thecomputing system 120 over the network (step 446). As shown in FIG. 4B,the computing system 120 receives the 3D vertices and performs sensordata fusion (step 404).

FIG. 4C also shows a process in which a dynamic sensor DSn can beactivated upon detection of an anomaly. The process is activated when ananomaly enters the operational area 330, or operational sub-region 320for which the sensor has observational perspective (Step 448). One ormore of the dynamic sensors DS1-DSn that have an observationalperspective of the operational area 330 detect the anomaly (step 450).Pixel morphology is performed on the detected anomaly and it isdetermined whether the morphology result is above a threshold (step452). If the morphology result is above the threshold, a 3D bounding boxis generated (step 454), object identification is performed using toanalyze the images within areas of detected movement (step 456), andpattern tracking is performed for the duration of the tracking of theanomaly (step 458). The one or more algorithms used for objectidentification are defined by one or more hierarchical models selectedfrom a defined list stored in memory or accessible over the network. Ifthe result of the morphology determination is below the threshold, thenanomaly detection is repeated (step 452). Following object detection(step 456), the dynamic sensor DSn determines whether any detectedobject morphology matches any model class by meeting or exceeding apredetermined percent threshold (step 460). If a threshold is met orexceeded, object recognition is performed to identify any uniqueidentifiers of the object (step 462) and another (e.g., second) 3Dbounding box is generated. If the detected object does not meet themodel class threshold, then the object detection operation is repeated(step 456). Following the object recognition operation (step 462), theprocessor of the dynamic sensor DSn determines whether the objectmatches any model class by meeting or exceeding a predetermined percentthreshold (step 464). If the predetermined percent threshold is met orexceeded, pattern tracking is performed (step 458) and another (e.g.,third) 3D bounding box is generated (step 454). Following patterntracking, the processor generates another (e.g., fourth) 3D boundingbox. After the 3D bounding box(es) is/are created, the dynamic sensorDSn communicates 3D vertices to the computing system 120 over thenetwork (step 460).

The computer program code for performing the specialized functionsdescribed herein can be stored on a medium and computer usable medium,which may refer to memories, such as the memory devices for both thecomputing system 120 and edge devices 114, which can be memorysemiconductors (e.g., DRAMs, etc.). These computer program products canbe a tangible non-transitory means for providing software to thecomputing system 100. The computer programs (e.g., computer controllogic) or software can be stored in the memory device. The computerprograms can also be received via the communications interface. Suchcomputer programs, when executed, can enable the computing system 120and edge device 114 to implement the present methods and exemplaryembodiments discussed herein. Accordingly, such computer programs mayrepresent controllers of the computing system 120 and edge devices 114.Where the present disclosure is implemented using software, the softwarecan be stored in a non-transitory computer readable medium and loadedinto the computing system 100 using a removable storage drive, aninterface, a hard disk drive, or communications interface, etc., whereapplicable.

The one or more processors of the computing system 120 and the edgedevices 114 can include one or more modules or engines configured toperform the functions of the exemplary embodiments described herein.Each of the modules or engines can be implemented using hardware and, insome instances, can also utilize software, such as program code and/orprograms stored in memory. In such instances, program code may becompiled by the respective processors (e.g., by a compiling module orengine) prior to execution. For example, the program code can be sourcecode written in a programming language that is translated into a lowerlevel language, such as assembly language or machine code, for executionby the one or more processors and/or any additional hardware components.The process of compiling can include the use of lexical analysis,preprocessing, parsing, semantic analysis, syntax-directed translation,code generation, code optimization, and any other techniques that may besuitable for translation of program code into a lower level languagesuitable for controlling the computer system 120 or edge device 114 toperform the functions disclosed herein. It will be apparent to personshaving skill in the relevant art that such processes result in thecomputer system 120 and/or edge device 114 being specially configuredcomputing devices uniquely programmed to perform the functions discussedabove.

It will be appreciated by those skilled in the art that the presentinvention can be embodied in other specific forms without departing fromthe spirit or essential characteristics thereof. The presently disclosedembodiments are therefore considered in all respects to be illustrativeand not restrictive. The scope of the invention is indicated by theappended claims rather than the foregoing description and all changesthat come within the meaning, range, and equivalence thereof areintended to be embraced therein.

What is claimed is:
 1. A system for monitoring an operational area, thesystem comprising: a sensor arrangement including two or more sensors,wherein: each sensor includes a processor configured to receive pluraldata streams from the sensor arrangement, each data stream including adifferent spatial characteristic of the operational area; and eachsensor is configured to: generate a three-dimensional (3D) virtualvisualization of the operational area based on observationalperspectives associated with the data streams and their associatedspatial characteristics; dynamically prioritize operational sub-regionswithin the operational area based on the spatial characteristics; andgenerate a signal encoded with data for verifying the 3D virtualvisualization of the operational area including the prioritizedoperational sub-regions.
 2. The system according to claim 1, furthercomprising: an interface that receives the signal from the processor andgenerates an output verifying the 3D virtual visualization of theoperational area.
 3. The system according to claim 1, wherein the two ormore sensors are configured to move between two or more locations ofobservational perspective with respect to the operational area.
 4. Thesystem according to claim 1, wherein the two or more sensors include anycombination of one or more sensor types.
 5. The system according toclaim 4, wherein each sensor comprises: at least one of an acoustic,radio frequency, electro-optical, thermal, chemical, biological,radiological, nuclear, explosive, temperature, or mechanical sensingcapability.
 6. The system according to claim 5, further comprising: acommunication interface for communicating with a network, and whereineach sensor is configured to: transmit a data stream periodically ornon-periodically to the processor over the network.
 7. The systemaccording to claim 6, wherein each sensor is configured to: transmitdata to the processor when a change in a spatial characteristic of anoperational sub-region is detected.
 8. The system according to claim 1,comprising: a memory device for storing the plural data streams.
 9. Thesystem according to claim 1, wherein each sensor is configured to:generate the 3D virtual visualization of the operational area bydividing the operational area into a plurality of 3D grid spaces. 10.The system according to claim 9, wherein each operational sub-regionincludes one or more 3D grid spaces.
 11. The system according to claim10, wherein each sensor is configured to: generate the plural datastreams by observing the operational area, wherein a 3D geometry of eachoperational sub-region is determined based on one or more sensingcharacteristics of the sensor arrangement.
 12. The system according toclaim 11, wherein the 3D geometry of each operational sub-region isdimensioned according to spatial properties of the operational area. 13.The system according to claim 11, wherein each sensor is configured to:dynamically change the 3D geometry of one or more operationalsub-regions based on a detected change in a spatial characteristic ofthe one or more operational sub-regions, the change in the spatialcharacteristic is determined from the plural data streams.
 14. Thesystem according to claim 10, wherein each sensor is configured to:assign a real-time prioritization value to each operational sub-regionbased on one or more of: an operational importance to a specifiedoperation, a spatial or temporal relevance to a specified operationactivity, the period since last sensor observation, availablegranularity of the spatial characteristics in the data streams, anobservational perspective of an operational sub-region for which thedata streams are generated, a sensor type, or velocity of an anomalywithin an operational sub-region of interest or a sub-region adjacent tothe sub-region.
 15. The system according to claim 14, wherein eachsensor is configured to assign on a case-by-case basis: a first priorityto an operational sub-region determined to be included in a securityoperation; a null priority to a sub-region determined to be outside of aspecified operational area; and a second priority to an operationalsub-region determined to be on a fringe of the operational area and/orundetectable by a sensor, the second priority being lower than the firstpriority.
 16. The system according to claim 14, wherein each sensor isconfigured to assign a priority to one or more operational sub-regions,or one or more sensors with observational perspective of an operationalsub-region, based on requirements of a specified operation, wherein atleast two or more of the assigned priorities are different.
 17. Thesystem according to claim 1, wherein each sensor is configured toprocess each data stream by detecting one or more of: a position of ananomaly in an operational sub-region or movement of an anomaly in anoperational sub-region.
 18. The system according to claim 17, whereineach sensor is configured to: extract portions of the spatialcharacteristics from each data stream and combine the extracted portionsinto a combined data set, and identify one or more patterns in thecombined data set.
 19. The system according to claim 1, wherein eachsensor is configured to: increase a priority of an operationalsub-region when an interval increases between receptions of data streamsfrom a sensor arrangement having observational perspective, if theduration of the interval exceeds a threshold.
 20. The system accordingto claim 1, configured to: assign a priority to an operationalsub-region based on the sensor arrangement's observational perspectiveof the sub-region, and the sensor arrangement's proximity to a detectedanomaly.
 21. The system according to claim 1, wherein each sensor isconfigured to: increase a priority of an operational sub-region overtime when data streams received from the operational sub-region areacquired by two or more sensor arrangements having a commonobservational perspective.
 22. The system according to claim 1, whereineach sensor has observational perspective of each operational sub-regionand the system is configured to: prioritize each sensor based oncharacteristics including a location of the operational sub-region, theinterval between data streams from the operational sub-regions, thelocation of an anomaly in or near the operational sub-region, or anobservational perspective of the operational sub-region.
 23. The systemaccording to claim 1, wherein the sensor arrangement comprises: a firstsensor arrangement and a second sensor arrangement, and the system isconfigured to: dynamically adjust the priority of a first operationalsub-region, adjust one or more properties of the first sensorarrangement based on the adjusted priority, and acquire data from thesecond sensor arrangement in a next highest priority operationalsub-region.
 24. The system according to claim 23, further comprising: acomputing system configured to: adjust one or more properties of thefirst sensor arrangement that has observational perspective of the firstoperational sub-region, to eliminate gaps in the sensor arrangement'sobservational perspective of the first operational sub-region, establishgranular observation of the first operational sub-region, or perform atriangulation of an anomaly under observation in the first operationalsub-region.
 25. The system according to claim 23, further comprising: acomputing system configured to: identify a first sensor of the firstsensor arrangement in the first operational sub-region that is currentlyengaged in a first observation activity, and identify a second sensor inthe first sensor arrangement that is available to engage in a secondobservation activity in the first operational sub-region.
 26. The systemaccording to claim 1, further comprising: a computing system configuredto: control one or more sub-systems for performing predefined criticaltasks to respond to or mitigate threats such as denial of serviceattacks or impairing or halting electronic functionality of the systemor devices associated with the system.
 27. The system according to claim26, in combination with the one or more subsystems, each of whichincludes an electronic warfare system, a directed energy system, or akinetic weapon system.
 28. A method for monitoring an operational area,the method comprising: in one or more devices connected in a network:receiving one or more data streams from a sensor arrangement thatincludes two or more sensors, each data stream including a differentspatial characteristic of the operational area; and in at least one ofthe two or more sensors of the sensor arrangement: generating a 3Dvirtual visualization of the operational area based on an observationalperspective associated with each data stream and the different spatialcharacteristics included in each data stream; dynamically prioritizingoperational sub-regions within the operational area based on thereceived spatial characteristics; and generating a signal for verifyingthe 3D visual virtualization of the operational area including theprioritized operational sub-regions.
 29. The method according to claim28, comprising: in an interface of the one or more devices or aninterface connected to the network: receiving the signal for verifyingthe 3D virtual visualization of the operational area; and generating anoutput that verifies the 3D virtual visualization of the operationalarea.
 30. The method according to claim 28, wherein generating the 3Dvirtual visualization of the operational area comprises: defining one ormore 3D grid spaces within the operational area.
 31. The methodaccording to claim 28, wherein each operational sub-region includes oneor more 3D grid spaces, the method comprising: defining a 3D geometry ofeach operational sub-region based on the observational perspective ofeach data stream and the spatial characteristics included in each datastream; and dynamically changing the 3D geometry of one or moreoperational sub-regions based on a detected change in a spatialcharacteristic of the one or more operational sub-regions.
 32. Themethod according to claim 31, comprising: assigning a real-time priorityto each operational sub-region based on one or more of: operationalimportance to a specified operation, spatial or temporal relevance to aspecified operation activity, the duration of the interval since lastsensor observation, available granularity of sensing, the observationalperspective of the sensor arrangement within an operational sub-region,the sensor-type, or velocity of an anomaly within an operationalsub-region or an adjacent operational sub-region.
 33. The methodaccording to claim 32, wherein assigning a real-time priority to eachoperational sub-region, on a case-by-case basis, comprises: assigning afirst priority to an operational sub-region that will be included insecurity operations; assigning a null priority to an operationalsub-region determined to be outside of a specified operational area; orassigning a second priority lower than the first priority to anoperational sub-region determined to be on a fringe of the operationalarea and/or undetectable by a sensor of the two or more sensors.
 34. Themethod according to claim 32, comprising: assigning a priority to one ormore operational sub-regions or one or more sensors having observationalperspective of an operational sub-region based on characteristics of aspecified operational activity, wherein two or more of the assignedpriorities are different.
 35. The method according to claim 34,comprising: identifying a first sensor in the sensor arrangement havingobservational perspective of the first operational sub-region that iscurrently engaged in a first observation activity, and identifying asecond sensor in the sensor arrangement that is available to engage in asecond observation activity in the first operational sub-region, whereinidentifying the second sensor in the first operational sub-region isbased on one or more of an availability of resources, a prioritizationof operational sub-regions, or a prioritization of anomalies.
 36. Themethod according to claim 34, comprising: dynamically adjusting thepriority of the first operational sub-region, and dynamically adjustingone or more properties of a first sensor arrangement in the firstoperational sub-region based on the adjusted priority and acquiringspatial characteristics of the first operational sub-region from asecond data stream received from a second sensor arrangement in a nexthighest priority operational sub-region.
 37. The method according toclaim 36, comprising at least one of: adjusting one or more propertiesof the sensor arrangement having observational perspective on the firstoperational sub-region to eliminate gaps in observational perspective incoverage of the 3D geometry of the first operational sub-region;establishing granular observation of the first operational sub-region;or performing a triangulation of an anomaly under observation in thefirst operational sub-region.
 38. The method according to claim 28,comprising: analyzing the data streams from a sensor arrangement havingobservational perspective of a first operational sub-region to detect aposition of an anomaly in the first operational sub-region, and/ormovement of an anomaly in the first operational sub-region.
 39. Themethod according to claim 38, comprising: extracting portions of thespatial characteristics from the data streams of the first operationalsub-region; combining the extracted portions into a combined data set;and identifying one or more patterns in the combined data set.
 40. Anon-transitory computer readable medium encoded with a program forperforming a method of monitoring an operational area, which when placedin communicative contact with a processor configures the processor to:generate a three-dimensional (3D) virtual visualization of theoperational area based on an observational perspective of plural datastreams from a sensor arrangement and different spatial characteristicsincluded in each data stream, wherein the sensor arrangement includestwo or more sensors, each sensor configured to: dynamically prioritizeoperational sub-regions within the operational area based on the spatialcharacteristics; and generate a signal for verifying the 3D virtualvisualization of the operational area including the prioritizedoperational sub-regions.
 41. A system for monitoring an operationalarea, the system comprising: a sensor arrangement including two or moresensors, each sensor including a processor configured to receive aplurality of data streams from the other sensors in the sensorarrangement, each data stream including a different spatialcharacteristic of the operational area, each sensor configured to:generate a three-dimensional (3D) virtual visualization of theoperational area based on observational perspectives associated with thedata streams and their associated spatial characteristics; dynamicallyprioritize operational sub-regions within the operational area based onthe spatial characteristics; increase a priority of one of theoperational sub-regions when an interval between reception of the datastreams from the sensor arrangement increases beyond a threshold; andgenerate a signal encoded with data for verifying the 3D virtualvisualization of the operational area including the prioritizedoperational sub-regions.
 42. A system for monitoring an operationalarea, the system comprising: a sensor arrangement including two or moresensors, each sensor including a processor configured to receive aplurality of data streams from the other sensors in the sensorarrangement, each data stream including a different spatialcharacteristic of the operational area, each sensor configured to:generate a three-dimensional (3D) virtual visualization of theoperational area based on observational perspectives associated with thedata streams and their associated spatial characteristics; dynamicallyprioritize operational sub-regions within the operational area based onthe spatial characteristics; assign a priority to one of the operationalsub-regions based on the observational perspective of the sensors in thesensor arrangement, proximity of the sensors in the sensor arrangementto a detected anomaly; and generate a signal encoded with data forverifying the 3D virtual visualization of the operational area includingthe prioritized operational sub-regions.
 43. A system for monitoring anoperational area, the system comprising: a sensor arrangement includingtwo or more sensors, each sensor including a processor configured toreceive a plurality of data streams from the other sensors in the sensorarrangement, each data stream including a different spatialcharacteristic of the operational area, each sensor configured to:generate a three-dimensional (3D) virtual visualization of theoperational area based on observational perspectives associated with thedata streams and their associated spatial characteristics; dynamicallyprioritize operational sub-regions within the operational area based onthe spatial characteristics; increase a priority of one of theoperational sub-regions over time when data streams received from theoperational sub-region are acquired by multiple sensor arrangementshaving a common observational perspective; and generate a signal encodedwith data for verifying the 3D virtual visualization of the operationalarea including the prioritized operational sub-regions.